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PREFACE 


This  document  is  an  annotated  version  of  a  briefing  to  be  given  at  the  Defense 
Analysis  Seminar  X  (DAS  X),  to  be  held  25-28  October  1999  in  Seoul,  Korea.  The 
seminar  is  sponsored  by  the  Deputy  Under  Secretary  of  the  Army,  Operations  Research, 
and  will  be  held  at  the  Korea  Institute  for  Defense  Analyses.  The  contents  of  the  briefing 
are  solely  the  opinions  of  the  author  and  do  not  represent  the  positions  of  either  the 
Institute  for  Defense  Analyses  or  the  U.S.  DoD. 
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Application  of 
Systems  Analysis  in  the 
U.S.  Department  of  Defense 

A  Presentation  at  DAS  X 
by 

Dr.  Stephen  3.  Balut 
October  1999 


Good  morning.  I’m  Steve  Balut,  the  Director  of  the  Cost  Analysis  and 
Research  Division  at  the  Institute  for  Defense  Analyses.  I  will  talk  about  the 
invention  of  systems  analysis  and  its  application  by  the  United  States  (U.S.) 
Department  of  Defense  (DoD). 
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Outline 

•  Leading  events 

•  Invention  and  rapid  expansion 

•  Weaknesses 

•  Curtailed  use 

•  Strengthened  process 

•  Reemergence 

•  Lessons 


This  slide  displays  an  outline  of  my  talk.  I  will  describe  the 
environment  in  which  systems  analysis  first  appeared,  grew  in  importance  and 
use,  then  stumbled,  and  eventually  recovered. 

Systems  analysis  was  invented  about  50  years  ago.  I  will  describe 
events  leading  to  its  invention  and  the  rapid  expansion  in  use  of  systems 
analysis  throughout  the  U.S.  DoD,  and  eventually  in  all  U.S.  federal  agencies. 
This  powerful  analytical  technique  was  implemented  rapidly  and  before  the 
knowledge  and  experience  necessary  to  its  successful  use  were  in  place.  As  a 
result,  some  early  systems  analyses  resulted  in  faulty  decisions  with  bad 
outcomes,  which  angered  the  U.S.  Congress  and  embarrassed  the  DoD.  The 
DoD  abruptly  curtailed  the  use  of  systems  analysis  until  the  needed  more 
knowledge  and  experience  were  gained.  About  10  years  ago,  a  strengthened 
process  reemerged  that  has  been  widely  applied  since.  I  will  close  by  telling 
you  about  lessons  we  learned  along  the  way.  These  lessons  might  be  useful  to 
you  as  you  implement  this  powerful  analytical  technique. 
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Definition  of  Terms 


A  "program"  is  a  multi-year  description  of  military 
capability  in  terms  of  resources  (people,  facilities, 
equipment,  and  dollars)  needed  to  accomplish  a 
defined  military  objective 


Before  going  any  further,  I  want  to  make  sure  you  understand  the 
meaning  of  a  term  that  I  will  use  throughout  this  presentation. 

The  term  “program”  can  mean  different  things  and  is  used  in  different 
ways.  In  this  discussion,  “program”  means  a  multi-year  description  of  military 
capabilities — in  terms  of  resources — that  are  planned  to  accomplish  a  defined 
military  objective.  Resources  include  people,  facilities,  equipment,  and  dollars. 
Said  another  way,  a  program  is  the  total  set  of  resources  planned  to  accomplish 
a  defined  set  of  military  functions. 

When  I  refer  to  the  “Navy  Program,”  for  example,  I  mean  the 
collection  of  all  people,  facilities,  equipment,  and  dollars  needed  to  perform 
the  missions  assigned  to  the  Navy.  When  I  refer  to  the  “Defense  Program,  I 
mean  the  aggregation  of  the  programs  from  each  Military  Department. 

Now,  on  with  the  presentation. 
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Leading  Events 
(1940s) 

•  Scientists  enlisted  to  help  with  war  effort 

•  Operations  research  invented 

•  After  war,  military  departments  preserved 
scientific  expertise 

•  "Think  tanks"  established 


Systems  analysis  was  invented  shortly  after  World  War  II.  Key  events 
during  and  after  the  war  created  the  demand  and  the  environment  that  gave 
birth  to  systems  analysis.  The  key  events  are  listed  on  this  slide. 

Shortly  after  the  start  of  the  war,  the  governments  of  the  Allied  Forces, 
particularly  the  United  Kingdom  and  the  United  States,  enlisted  scientists  to 
aid  the  war  effort.  These  scientists  studied  the  operations  of  war  in  a 
quantitative,  scientific  way.  These  important  studies  laid  the  groundwork  for 
the  later  invention  of  systems  analysis. 

The  objective  was  to  improve  the  efficiency  of  operations  in  progress 
or  planned  for  the  future.  This  line  of  study  came  to  be  called  “operations 
research.”  Operations  research  sought  no  more  than  to  do  something  better,  to 
use  scientific  methods  to  get  the  most  out  of  available  resources. 

During  the  war,  the  U.S.  Military  Departments  had  well-defined  and 
well-understood  missions — the  Army’s  mission  was  land  warfare;  the  Navy’s 
mission  was  power  projection  from  the  sea,  and  the  Army  Air  Corps’s  mission 
was  air  superiority,  bombing,  and  combat  support.  But  after  the  war,  the 
distinction  between  mission  responsibilities  became  blurred  and  eventually 
broke  down.  This  was  due  to  both  the  rapid  development  of  military 
technology  and  the  evolving  character  of  the  military  opposition.  Inter-service 
competition  for  missions  became  fierce.  President  Harry  S.  Truman  tried  to 
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limit  this  competition.  In  1947,  the  U.S.  DoD  and  Joint  Chiefs  of  Staff  (JCS) 
were  established.  With  the  establishment  of  the  DoD,  a  cabinet-level  office 
headed  by  a  civilian  was  given  the  responsibility  for  allocating  defense  budgets 
to  the  military  services.  The  JCS  assisted  the  new  Secretary  of  Defense  identify 
which  missions  would  be  performed  by  each  service. 

During  and  after  the  war,  the  military  departments  took  steps  to 
institutionalize  and  preserve  the  scientific  talents  and  resulting  operations 
research  capabilities  that  were  brought  to  bear  so  successfully  on  the  war  effort. 
To  do  this,  several  defense  research  and  analysis  organizations  so-called 
“think  tanks” — were  established. 

Three  groups  of  scientists  that  had  worked  on  Navy  problems  were  later 
consolidated  to  form  the  Center  for  Naval  Analyses  (CNA).  In  1947,  the  Army 
Air  Corps  established  the  RAND  Corporation.  RAND’s  purpose  was  “to 
maintain  scientific  expertise... and  to  conduct  independent  and  objective 
national  security  research....”  At  about  the  same  time,  the  Weapon  Systems 
Evaluation  Group  was  formed  to  assist  the  newly  formed  JCS.  The  group’s 
purpose  was  “...to  perform  technical  analyses  of  service  and  multi-service 
weapon  systems.”  The  Institute  for  Defense  Analyses  (IDA),  the  research 
organization  I  represent,  was  established  several  years  later  to  augment  and 
eventually  take  over  and  expand  on  the  Weapon  Systems  Evaluation  Group  s 
mission.  Additional  organizations  were  formed,  both  within  and  outside  the 
government,  including  professional  scientific  societies,  to  preserve  and  build  on 
the  expertise  in  operations  research  that  existed  at  the  end  of  the  war. 
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Invention  of  Systems  Analysis 
(Late  1940s) 

•  Systems  analysis  grew  out  of  operations 
research 

•  Economic  principles  applied  to  Air  Force 
decisions 

•  Cost-effectiveness  (CE)  criterion  of  choice 

•  Cost  analysis  invented  to  support  CE 


In  the  late  1940s,  the  concept  of  conducting  analysis  from  the 
perspective  of  the  weapon  systems  being  used  grew  naturally  out  of  operations 
research.  The  original  term  “weapon  systems  analysis,”  first  coined  at  the 
RAND  Corporation  in  1948,  was  quickly  shortened  to  “systems  analysis.” 

Systems  analysis  sought  (and  still  seeks)  to  do  all  that  operations 
research  did — to  use  scientific  methods  to  do  something  better,  getting  the 
most  out  of  available  resources — but  it  also  sought  to  do  the  right  thing  better 
and  more  cheaply.  Systems  analysis  thus  sought  to  answer  a  more  creative 
question  than  did  operations  research.  It  was  oriented  more  toward  the  future 
and  had  a  broader  outlook. 

RAND’s  early  systems  analyses  for  the  Air  Force  integrated  economic 
principles  with  operations  research.  The  basic  analytical  question  of  “Which 
system  is  best  for  the  job?”  changed  to  “Given  a  fixed  budget,  which  system  is 
most  cost-effective?”  A  corollary  is  “Given  a  fixed  level  of  effectiveness, 
which  system  can  do  the  job  most  cheaply?”  The  use  of  dollar  costs  in  these 
studies  as  a  proxy  for  real,  economic  costs  marked  the  birth  of  “cost- 
effectiveness  analysis,”  and  cost  analysis  was  a  crucial  component  of  the  cost- 
effectiveness  equation. 

Cost-effectiveness  was  the  heart  and  soul  of  systems  analysis  from  the 
start.  It  was  the  criterion  of  choice. 
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Let’s  take  a  quick  look  at  an  early  application  of  systems  analysis  to 
observe  one  of  its  early  problems. 

In  1949,  RAND  conducted  the  first  large  systems  analysis  for  the  Air 
Force.  The  analysis  compared  the  prospective  B-52  bomber  to  a  turbo-prop 
alternative.  Because  RAND  did  not  yet  have  a  cost  analysis  capability,  the  Air 
Force  provided  RAND  with  the  costs  for  both  alternatives.  RAND’s  analysis 
favored  the  turbo-prop,  but  the  Air  Force  clearly  wanted  to  buy  the  B-52.  After 
RAND  briefed  its  results  to  the  Air  Force,  the  Air  Force  dropped  the  cost  of  the 
B-52  by  half  and  increased  the  cost  of  the  turbo-prop  by  half.  Needless  to  say, 
these  changes  tilted  the  analysis  in  favor  of  the  B-52.  The  rest  is  history;  the 
B-52  is  still  in  the  U.S.  arsenal. 

Some  analysts  at  RAND  felt  they  had  been  intentionally  deceived  by  the 
Air  Force.  As  a  result,  RAND  established  its  own  cost  analysis  department  to 
ensure  an  independent  cost  analysis  capability.  Through  the  early  1950s,  RAND 
analysts  invented  and  refined  the  building  blocks  of  cost  analysis  that  continue 
to  serve  us  today:  the  use  of  “one-time”  and  “recurring”  costs,  cost  estimating 
relationships,  parametric  cost  estimating,  incremental  costing,  and  force  costing. 
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Contributing  Factors 
(Late  1940s  and  1950s) 

•  RAND  develops  a  Planning,  Programming  and 
Budgeting  system  for  the  Air  Force 

•  Separation  of  service  missions  blurred 

•  Budget  consciousness  causes  fierce  competition  for 
resources 

•  DoD  moves  toward  management  by  mission 

•  Economic  principles  not  used  to  allocate  resources  by 
the  Office  of  the  Secretary  of  Defense 

•  Civilian  administration  losing  control  of  allocation  of 
resources  by  mission 


The  use  of  the  new  analytical  technique  called  systems  analysis 
expanded  rapidly.  In  retrospect,  it  may  have  expanded  too  rapidly.  The  reasons 
for  its  speedy  acceptance  and  use  had  to  do  with  the  environment  at  the  time  in 
the  U.S.  DoD. 

In  the  early  1950s,  RAND  developed  a  new  expense  accounting  system 
for  the  Air  Force.  The  new  idea  involved  associating  dollars  with  missions, 
(defense  outputs)  rather  than  functions  (defense  inputs)  such  as  ammunition 
and  fuel.  This  change  led  to  a  whole  new  approach  to  planning,  programming, 
and  budgeting  resources  for  war. 

Up  to  this  point,  the  procedure  for  funding  the  U.S.  defense  program 
was  as  follows:  Congress  set  aside  an  amount  for  national  security,  the 
Secretary  of  Defense  divided  it  up  among  the  services,  and  the  services 
planned  how  to  spend  their  allotted  dollars  on  people,  facilities,  equipment, 
and  operations  to  carry  out  their  assigned  missions. 

This  procedure  worked  well  as  long  as  the  services  had  clearly  defined 
and  separated  missions,  which  was  the  case  through  the  end  of  World  War  II. 
However,  during  the  late  1940s  and  1950s,  the  distinction  between  these 
missions  became  blurred  by  rapid  expansion  of  new  technologies,  such  as 
nuclear  power,  and  by  the  new  threats  posed  by  the  different  world  order.  It 
became  increasingly  clear  that  the  procedure  used  to  allocate  defense  resources 
would  not  work  without  a  strong  separation  of  missions. 
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Exacerbating  the  problem,  President  Dwight  D.  Eisenhower  reduced  the 
amount  of  money  spent  on  national  security  during  the  1950s.  Smaller  defense 
budgets  created  fierce  competition  for  resources  among  the  services. 

To  rectify  the  problem,  the  DoD  took  a  few  tentative  steps  away  from 
management  by  service  and  towards  management  by  mission.  Unified 
commands  were  established  on  the  basis  of  a  mission  rather  than  a  theater. 
Unified  commanders  were  given  primary  responsibility  for  operational  control 
of  associated  forces. 

But  while  the  application  of  economic  principles  to  defense  decisions 
was  beginning  (thanks  to  RAND)  in  the  Air  Force,  the  rest  of  the  DoD  had  not 
yet  embraced  the  new  technique. 

By  the  late  1950s,  civilian  control  over  the  military  had  clearly  began  to 
weaken.  Because  of  the  practice  of  allocating  dollars  to  the  services  when 
service  missions  were  not  separate,  civilians  in  the  Office  of  the  Secretary  of 
Defense  had  been  losing  control  of  the  central  political  element  of  the  defense 
program — the  allocation  of  resources  among  missions.  One  solution  proposed 
and  rejected  was  to  reorganize  the  services  along  mission  lines.  Another 
solution,  proposed  by  RAND  in  1954,  1956  and  again  in  1960,  was  to  change 
program  control  from  a  service  to  a  mission  basis.  Under  this  proposal,  the 
services  would  compete  with  each  other  for  major  missions. 
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Application  Expanded 
(Early  1960s) 

•  PPBS  implemented  DoD-wide 

•  Use  of  systems  analysis  quickly  expanded 

•  Military  departments  ill-prepared 

•  Low-cost  systems  won  budget  battles 

•  Use  expanded  to  all  federal  agencies 

•  Organizational  changes  institutionalized 
process 


In  1960,  a  book  was  published  that  led  to  changes  in  the  way  the  DoD 
did  business.  The  authors  were  Charles  Hitch  and  Roland  McKean,  both  of  the 
RAND  Corporation.  The  book,  Economics  for  Defense  in  the  Nuclear  Age, 
described  how  to  apply  principles  of  economics  to  government  (especially 
defense)  decisions.  Derived  from  RAND’s  development  and  application  of 
systems  analysis,  the  book  addressed  economic  efficiency  and  stressed  the 
need  to  compare  costs  and  effectiveness.  It  also  provided  counter  arguments  to 
detractors.  For  example,  here  is  a  quote:  “An  economically  efficient  solution 
to  military  problems  does  not  imply  a  cheap  force  or  a  small  military  budget.  It 
simply  implies  that  whatever  the  military  budget,  the  greatest  military 
capabilities  are  developed.” 

In  1961,  President  John  F.  Kennedy  appointed  Robert  McNamara 
Secretary  of  Defense,  who  appointed  Charles  Hitch,  one  of  the  authors  of  the 
book,  as  DoD  Comptroller.  Hitch  immediately  established  the  position  of 
Director  for  Systems  Analysis,  and  appointed  another  RAND  colleague,  Alain 
Enthoven  as  the  head.  Enthoven,  like  Hitch,  was  part  of  the  small  group  of 
analysts  who  had  been  developing  the  systems  analysis  technique  at  RAND 
over  the  previous  10  years. 
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Thus  began  a  period  in  the  DoD  of  rational  and  more  centralized 
decision  making  under  constrained  budgets.  Defense  Comptroller  Hitch 
initiated  a  defense-wide  Planning,  Programming  and  Budgeting  System 
modeled  after  the  Air  Force’s  system.  To  quote  the  implementing  directive, 
“the  ultimate  objective  of  the  Planning,  Programming,  and  Budgeting  System 
shall  be  to  provide  operational  Commanders-in-Chief  the  best  possible  mix  of 
forces,  equipment,  and  support  attainable  within  fiscal  constraints.” 

The  defense  program  structure  was  immediately  changed  from  being 
oriented  around  inputs  (lists  of  things  to  buy,  such  as  fuel  and  ammunition)  to 
outputs  (capabilities  to  perform  missions,  such  as  conventional  warfare  and 
strategic  warfare). 

The  new  Systems  Analysis  Office  began  examining  proposed  weapon 
systems  in  a  systematic,  rational  manner  using  cost-effectiveness  to  support 
decisions.  This  early  period  in  the  development  of  basic  cost  analysis 
capabilities  was  marked  by  rapid  expansion  of  financial  management  systems. 

Due  to  the  speedy  introduction  of  this  new  criterion  called  cost- 
effectiveness,  the  services  were  not  well  prepared  to  present  or  defend  their 
programs  to  the  Office  of  the  Secretary  of  Defense.  The  services  learned 
quickly  that  low  cost  systems  won  budget  battles,  and  high-cost  systems  were 
unlikely  to  be  approved. 

In  1965,  President  Lyndon  B.  Johnson  directed  all  federal  agencies  to 
implement  a  Planning,  Programming  and  Budgeting  System.  The 
implementing  memorandum  from  the  Bureau  of  the  Budget  directed  that 
analysis  comparing  the  benefits  and  costs  of  alternative  programs  be  carried 
out  before  budget  approval  could  be  obtained. 

Also  in  1965,  the  Systems  Analysis  Office  within  the  Office  of  the 
Secretary  of  Defense  was  elevated  to  the  level  of  an  Assistant  Secretary  of 
Defense.  An  economics  division  and  a  cost  analysis  division  were  established 
to  improve  the  department’s  capabilities  in  those  areas.  By  then,  the  Office  of 
the  Secretary  of  Defense  had  been  conducting  systems  analyses  for  5  years 
without  benefit  of  these  offices. 
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Weaknesses 
(Late  1960s) 

•  Bad  experiences 

•  Cost  overruns 

•  Military  leaders  spoke  out  against  systems 
analysis 

•  Limitations  explained  to  Congress 


Many  important  decisions  were  made  in  the  early  1960s  based  on  the 
results  of  systems  analyses,  most  of  which  were  conducted  by  the  Office  of  the 
Secretary  of  Defense.  Within  a  few  years,  it  became  clear  in  retrospect  that 
some  of  those  decisions  had  been  bad  ones.  It  also  became  clear  that  the  DoD 
had  rushed  into  the  use  of  systems  analysis  without  adequate  preparation.  That 
is,  the  tools  required  to  conduct  systems  analysis  successfully  were  not  yet 
available.  These  included  data  and  methods  for  forecasting  both  costs  and 
effectiveness  and  people  knowledgeable  and  experienced  in  the  use  of  systems 
analysis. 

The  services  continued  to  lose  budget  battles  through  the  end  of  the 
1960s.  The  Skybolt  missile  and  B-70  bomber  were  canceled.  The  Minuteman 
missile  procurement  was  reduced.  The  battle  for  NIKE  missile  program  was 
lost.  Navy  escort  ships  were  not  approved.  Approval  of  the  Polaris  submarine 
and  missile  system  ended  Air  Force  dominance  of  nuclear  weapons.  Some 
systems  were  forced  on  the  services:  the  M-16  rifle  was  forced  on  the  Army, 
and  the  F-4  fighter  aircraft  was  forced  on  the  Air  Force. 

By  1969,  twenty-seven  out  of  thirty-five  major  systems  were  exhibiting 
substantial  cost  growth.  The  DoD’s  ability  to  produce  reliable  cost  estimates 
came  into  question,  and  Congress  became  less  tolerant  of  cost  overruns. 
Observers  also  noted  poor  choices  of  measures  of  effectiveness  by  analysts  in 
the  Systems  Analysis  Office.  In  some  cases,  the  measures  of  effectiveness 
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used  did  not  accurately  reflect  the  true  objectives  of  the  analyses,  resulting  in 
biased  and  flawed  findings  and  recommendations. 

High-ranking  military  leaders,  angry  over  the  cost  overruns,  started 
speaking  out  in  public  against  systems  analysis.  Here  are  two  examples  of 
what  was  said.  General  Ira  Eaker:  “one  of  the  prime  obstacles  to  adequate 
defense  weapons... has  been  a  hurdle  called  cost-effectiveness.  This  test 
applied  by  scientists  and  theorists  has  killed  off  many  new  weapons  urgently 
needed  by  military  leaders.” 

Another  quote,  by  Admiral  Hyman  Rickover:  “...decisions  appear  to 
be  made  by  the  rules  that  are  ground  out  by  the  cost-effectiveness  analysts.  It 
never  was  the  intent  of  the  defense  unification  act  that  a  group  of  analysts 
would,  in  effect,  become  the  decision-making  apparatus  of  our  defense 
establishment.  ...  just  what  are  the  qualifications  of  the  cost  analysts?  Their 
experience?  Their  accomplishments?  Their  reputations?  ...the  only  record  of 
accomplishments  of  these  social  scientists  is  reducing  budgets.” 

In  1968,  Secretary  of  Defense  Schlessinger  warned  Congress  of  the 
limitations  of  economic  analysis  in  national  security  issues  in  his  memo  titled 
“Uses  and  Abuses  of  Analysis.” 
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Curtailed  Use 
(1970s) 

•  DoD  decision  making  decentralized 

•  Systems  Analysis  Office  downgraded;  name 
and  role  changed 

•  Term  "economic  analysis"  substituted  for 
"systems  analysis" 


After  nearly  a  decade  of  use,  including  some  abuse,  systems  analysis 
earned  a  bad  reputation,  fell  from  favor,  and  was  reeled  in.  Cost-effectiveness 
had  been  misused,  sometimes  out  of  ignorance,  other  times  intentionally.  The 
absence  of  tools  to  produce  credible  estimates  made  forecasting  tenuous  and 
policing  of  the  process  almost  impossible.  Something  had  to  change  in  the  way 
defense  decisions  were  made  to  regain  credibility  with  Congress  and  the 
American  people. 

In  1969,  the  Secretary  of  Defense  made  both  procedural  and 
organizational  changes  to  de-emphasize  the  role  of  systems  analysis  at  the 
level  of  the  Office  of  the  Secretary  of  Defense  and  return  decision-making 
authority  to  the  services.  Rather  than  using  systems  analysis  to  initiate  plans 
and  systems,  analysts  in  the  Office  of  the  Secretary  instead  were  to  passively 
“review”  programs  proposed  by  the  services.  The  name  of  the  Secretary’s 
Systems  Analysis  Office  was  changed  to  “Program  Review  and  Evaluation”  to 
correctly  describe  their  new  role.  The  term  “economic  analysis”  was 
substituted  for  “systems  analysis”  in  key  DoD  directives.  The  effect  of  these 
changes  was  to  shift  the  practice  of  systems  analysis  from  the  Office  of  the 
Secretary  to  the  military  departments.  And,  while  the  name  was  different,  the 
procedure  remained  the  same. 
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Strengthened  Process 
(1970s  and  1980s) 

•  Resolve  to  fix  procurement  ills 

•  Milestones  established 

•  Services  directed  to  improve  cost  analysis 
capabilities 

•  Cost  data  collection  systems  initiated 

•  New  rules/policies  with  regard  to  cost 

•  Cost  organizations  established 


Starting  in  the  early  1970s,  defense  managers  took  aggressive  steps  to 
fix  the  problems  with  defense  procurement  that  they  believed  were  caused  by 
faulty  systems  analyses.  These  managers,  determined  to  stop  cost  overruns, 
focused  on  improving  capabilities  to  develop  credible  cost  estimates.  Major 
milestones  were  established  for  acquisition  programs  and  cost  reviews  were 
required  at  each  milestone.  The  Cost  Analysis  Improvement  Group — or  CAIG, 
as  it  is  called — was  established  in  the  Office  of  the  Secretary  of  Defense  and 
independent  cost  estimates  were  required  of  this  new  group  at  the  same  time 
the  services  presented  their  cost  estimates  at  milestone  reviews.  The  services 
were  directed  to  improve  their  cost  analysis  capabilities.  Cost  data  collection 
systems  were  initiated  to  provide  cost  analysts  with  the  information  they 
needed  to  reduce  error  in  their  estimates. 

In  the  early  1980s,  Deputy  Secretary  of  Defense  Carlucci  initiated  a 
long  list  of  actions  aimed  at  improving  the  department’s  ability  to  forecast 
future  costs.  These  actions  included  requiring  explicit  consideration  of  cost 
risk  and  uncertainty,  budgeting  to  most  likely  or  expected  costs,  use  of 
economic  production  rates,  allowing  multi-year  procurement  based  on 
benefit/risk  analysis,  and  budgeting  more  realistically  for  inflation. 

In  1984,  Congress  directed  the  Secretary  of  Defense  to  consider  an 
independently  derived  life-cycle-cost  estimate  before  approving  either  the 
development  or  production  of  any  new  weapon  system.  In  response,  the 
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Secretary  of  Defense  provided  resources  to  establish  cost  analysis  centers  and 
agencies  in  each  of  the  military  departments  and  required  them  to  produce 
these  independent  estimates. 
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Reemergence 

(1990s) 

•  Decisions  on  major  issues  supported  by  cost- 
effectiveness  studies 

•  COEA's  required  by  DoD  Directive 

•  Now  called  "analysis  of  alternatives"  (AOA) 


With  time  and  concerted  effort,  the  U.S.  DoD’s  abilities  to  conduct 
cost-effectiveness  analyses  improved  and  its  use  expanded.  The  department 
invested  in  improving  its  cost  analysis  skills,  and  these  investments  paid  off. 
Research  sponsored  and  conducted  by  cost  analysis  organizations  in  the  Office 
of  the  Secretary  and  in  the  services  improved  both  the  data  and  methods  used 
to  forecast  future  costs.  These  research  efforts  were  supported  by  contractors 
like  RAND  and  IDA.  The  size  of  cost  overruns  declined  as  these  capabilities 
improved. 

At  about  this  time,  responsibility  for  conducting  independent  cost 
estimates  for  Major  Weapon  System  Milestone  Reviews  shifted  from  the 
military  departments  to  the  Office  of  the  Secretary  of  Defense.  The  CAIG  now 
develops  and  presents  these  independent  estimates,  and  the  military 
departments  now  present  what  is  referred  to  as  “Service  Cost  Positions.” 

Through  the  late  1980s  and  the  1990s,  large-scale  systems  analyses 
were  increasingly  conducted  with  the  help  of  contractors,  such  as  IDA,  RAND, 
and  CNA,  who  have  special  relationships  with  government  offices. 

Starting  in  1991,  systems  analyses  were  required  by  DoD  Directive,  but 
under  yet  another  name.  The  new  name,  “cost  and  operational  effectiveness 
analysis,”  was  a  much  more  descriptive  term  than  “economic  analysis,”  which 
had  been  substituted  for  “systems  analysis”  12  years  earlier. 
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More  recently — within  the  past  few  years — systems  analysis  has  been 
practiced  at  all  levels  within  the  DoD  under  the  name  “analysis  of 
alternatives.”  In  these  most  recent  studies,  analysts  representing  the  Office  of 
the  Secretary  of  Defense,  the  military  departments,  and  contractors  collaborate 
to  conduct  the  analyses. 


18 


Lessons 

Systems  analysis  is  a  powerful  technique  for 
aiding  public  sector  decisions;  however,... 

-  Building  blocks  must  be  in  place  before 
implementation 

•  Cost  data  and  methods 

•  Tools  to  conduct  effectiveness  studies 

•  People  trained/experienced  in 
application 

-  Must  guard  against  misuse 


Looking  back,  we  can  all  learn  from  the  U.S.  experience.  Systems 
analysis  is  unquestionably  a  powerful  tool  for  shedding  light  on  the  most 
important  issues  associated  with  a  decision  to  allocate  public  funds. 
Government  decision-makers  are  better  prepared  to  make  good  choices  after 
considering  both  the  costs  and  effectiveness  of  alternatives.  However,  the  U.S. 
experience  clearly  demonstrates  that  arithmetic  can  be  misused  and  abused, 
leading  to  faulty,  costly  decisions.  We  have  learned  that  analysts  cannot 
conduct  credible  systems  analyses  without  first  doing  some  homework.  That 
is,  certain  building  blocks  should  be  in  place  before  applying  the  technique. 

One  of  the  most  troublesome  areas  has  been  costs.  When  systems 
analysis  was  first  applied,  cost  analysis  had  not  yet  been  invented.  The  DoD 
did  not  know  how  to  forecast  future  costs  of  weapon  systems,  and  that  ability 
was  developed  slowly.  Along  the  way,  many  bad  cost  estimates  made  their 
way  into  cost-effectiveness  studies.  Eventually,  actual  cost  experiences 
showed  the  poor  quality  of  these  estimates.  This  practice  resulted  in 
embarrassing  and  destructive  cost  overruns  that  robbed  future  budgets  to  pay 
off  debts  that  had  grown  larger  than  planned. 

The  other  troublesome  area  was  effectiveness.  Measures  of 
effectiveness  used  were  sometimes  faulty,  and  models  for  projecting  the 
effectiveness  of  future  systems  had  yet  to  be  developed. 
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A  more  rational  approach  would  have  been  to  develop  at  least  basic 
capabilities  to  forecast  both  cost  and  effectiveness  before  implementing 
systems  analysis  throughout  the  department.  These  basic  capabilities  would 
include  data,  methods,  and  tools,  along  with  people  trained  and  experienced  in 
their  use. 

Given  fundamental  skills,  tools,  and  information,  the  Office  of  the 
Secretary  would  have  been  better  prepared  to  police  the  practice,  making  sure 
that  systems  analysis  was  not  misused. 

That  concludes  my  presentation. 
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